Deep Learning with Apache SystemML

Deep Learning (DL) is a subfield of Machine Learning (ML) that focuses on learning hierarchical representations of data with multiple levels of abstraction using neural networks [15]. Recent advances in deep learning are made possible due to the availability of large amounts of labeled data, use of GPGPU compute, and application of new techniques (such as ReLU, batch normalization [12], dropout [17], residual block [10], etc.) that help deal with issues in training deep networks. In spite of the need to train on large datasets, there is a disconnect between the deep learning community and the big data community. To scale to a multi-node cluster, most deep learning frameworks (such as Caffe2, TensorFlow [2] and IBM’s PowerAI DDL [6]) use custom communication libraries based on either MPI (such as IBM SpectrumMPI, Facebook’s Gloo) or a customnetworking protocol (such as Google RPC). Unlike popular big data frameworks (such as Apache Hadoop [9] and Apache Spark [18]), these communication libraries do not provide features such as resource sharing, multi-tenancy and fault-tolerance out of the box, making them difficult to deploy on shared production clusters. Œis leads to ineffective use of resources in an organization, o‰en requiring two separate infrastructures (i.e. scale-up versus scale-out). Œis problem is even more severe when the data generated as part of the big data pipeline (ML, data preprocessing, data cleaning) needs to be consumed by the deep learning pipeline or vice versa, as the workload characteristics of a typical machine learning algorithm (i.e. memory-bound, BLAS level-2, sparse/ultrasparse inputs (or feature matrix), etc.) are o‰en different than that of a typical deep learning algorithm (i.e. compute-bound, BLAS level-3, dense inputs, etc.). Apache SystemML [4] aims to bridge that gap by seamlessly integrating with underlying big data frameworks and by providing a unified framework for implementing machine learning and deep learning algorithms. In Apache SystemML, the ML algorithms are implemented using a high-level R-like language called DML (short for Declarative Machine Learning). DML improves the productivity of data scientists by enabling them to implement their ML algorithm with precise semantics as well as abstract data types and operations, independent of the underlying data representation or cluster characteristics. For the given DML script, SystemML’s cost-based compiler automatically generates hybrid runtime execution plans that are composed of single-node and distributed operations depending on data and cluster characteristics such as data size, data sparsity, cluster size and memory configurations, while exploiting the capabilities of underlying data-parallel frameworks such asMapReduce or Spark. Œis allows for algorithm reusability across data-parallel frameworks, and simplified deployment for varying data characteristics and runtime environments, ranging from low-latency scoring to large-scale training. 2 DEEP LEARNING APIS